An Overview of High Order Reverse Mode

Automatic Differentiation (AD) is increasingly an important component of Machine Learning (ML) packages. For evaluating the gradient, the first order reverse mode also known as back-propagation, is optimal and is widely used. However, the functionalities of current mainstream ML packages for evaluating second and higher order derivatives are limited. One reason is that high order derivatives are computed using an overlay of first order forward and reverse modes. Here we describe an algorithm and code that directly implements the high order reverse mode, called ReverseAD. It is more efficient than previous methods, especially when evaluating sparse derivatives. 1 High Order Reverse Mode of Automatic Differentiation